Low-rank matrix factorization (LRMF) is a canonical problem in non-convex optimization, the objective function to be minimized is non-convex and even non-smooth, which makes the global convergence guarantee of gradient-based algorithm quite challenging.

Neural Information Processing Systems http://nips.cc/

In recent years, transformer-based models have revolutionized deep learning, particularly in sequence modeling.

Neural Information Processing Systems http://nips.cc/

SGD until 100% accuracy is achieved, in four different settings: 1. Random initialization + Training with true labels.

Learning with deep neural networks has enjoyed huge empirical success in recent years across a wide variety of tasks.

Jastrz ebski et al. [6] suggested that the ratio between the learning rate and the batch

The question of which global minima are accessible by a stochastic gradient decent (SGD) algorithm with specific learning rate and batch size is studied from the perspective of dynamical stability. The concept of non-uniformity is introduced, which, together with sharpness, characterizes the stability property of a global minimum and hence the accessibility of a particular SGD algorithm to that global minimum. In particular, this analysis shows that learning rate and batch size play different roles in minima selection. Extensive empirical results seem to correlate well with the theoretical findings and provide further support to these claims.